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Objective: Effective response to biological events necessitates ongoing evaluation of pre- 
paredness. This study was a bilateral German-Israeli collaboration aimed at developing an 
evaluation tool for assessing preparedness of medical facilities for biological events. 

Methods: Measurable parameters were identified through a literature review for inclu- 
sion in the evaluation tool and disseminated to 228 content experts in two modified Delphi 
cycles. Focus groups were conducted to identify psychosocial needs of the medical teams. 
Table-top and functional exercises were implemented to review applicability of the tool. 

Results: One hundred seventeen experts from Germany and Israel participated in the 
modified Delphi. Out of 188 parameters that were identified, 183 achieved a consensus of 
>75% of the content experts. Following comments recommended in the Delphi cycles, 
and feedback from focus groups and hospital exercises, the final tool consisted of 172 para- 
meters. Median level of importance of each parameter was calculated based on ranking 
recommended in the Delphi process. Computerized web-based software was developed 
to calculate scores of preparedness for biological events. 

Conclusion: Ongoing evaluation means, such as the tool developed in the study, can 
facilitate the need for a valid and reliable mechanism that may be widely adopted and 
implemented as quality assurance measures. The tool is based on measurable parame- 
ters and indicators that can effectively present strengths and weaknesses in managing a 
response to a public health threat, and accordingly, steps can be implemented to improve 
readiness. Adoption of such a tool is an important component of assuring public health and 
effective emergency management. 

Keywords: standard operating procedures, measurable indicators, biological event, evaluation of emergency 
preparedness, disaster management 



INTRODUCTION 
BACKGROUND 

Biological agents do not recognize borders ( 1 ) and often spread 
across continents, causing great concern to governments and the 
public (2). The various outbreaks that occurred in the past decade 
such as the severe acute respiratory (3), the avian flu (4), the 
A/H1N1 pandemic (5), as well as the potential threat of bio ter- 
ror events, emphasize the need for countries to work together to 
combat public health crises. A recent study conducted in Europe 
focusing on prevention and control of communicable diseases 
stressed the need to prioritize awareness of communicable dis- 
eases and provide guidance for best practices, in order to facilitate 
an evidence-based improvement of emergency preparedness (6). 



The increased risk of world-wide infectious disease spread has 
led to the understanding that a regional and even global response 
is needed (7). As local and national response impacts on the overall 
emergency response, collaboration between neighboring as well as 
distant countries is crucial in achieving preparedness. 

Inter-country and global collaboration is orchestrated by the 
World Health Organization (WHO) through the revised Inter- 
national Health Regulations (7, 8). WHO promotes the fol- 
lowing main components in planning and managing public 
health events: ongoing surveillance of mortality and morbidity; 
early detection of unexpected, potential internationally spread- 
ing biological events; and, continuous assessment of emergency 
preparedness (8). 
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Medical facilities are vital components in an effective response 
to biological events (9). Hospitals and emergency medical ser- 
vices consistently coordinate their work and maintain close col- 
laborations, but primary care institutions often do not partic- 
ipate in emergency-preparedness coalitions (10). Studies have 
presented that, though there is an increase in appropriate emer- 
gency operation plans, information-sharing technologies, new 
equipment, communication and surveillance systems, as well as 
enhanced preparedness training programs, there is nevertheless 
low confidence of medical teams in managing specific biological 
incidents (2, 11). 

Ensuring an effective response to a potential public 
health/biological event necessitates adoption of evaluation mech- 
anisms. The evaluation enables to identify strengths in capacities 
and competences that should be sustained and weaknesses that 
must be rectified (12). In addition, evaluation of disaster pre- 
paredness is crucial in order to strengthen national healthcare 
systems and achieve resilience of health institutions (13). Vali- 
dated preparedness measures should be applied in order to support 
accountability and improved outcomes in maintaining emergency 
preparedness and management (14, 15). 

Extensive efforts are made to develop State of the Art means 
for evaluating emergency preparedness of healthcare systems for 
the various scenarios. Exercises simulating potential scenarios are 
used to identify public health systems-level challenges (11). Most 
exercises are time and resource consuming, and therefore are not 
implemented as frequently as recommended. Other evaluation 
means are needed to assure creation and sustenance of emergency 
preparedness for biological events (16). 

To date, there is a lack of a widely approved and adopted 
evaluation mechanism that facilitates a continuous assessment of 
preparedness for biological events. 

The present study was initiated as a bilateral German-Israeli 
quest to develop an evaluation tool that will enable to assess pre- 
paredness of health services and public healthcare professionals 
for biological events, and promote emergency preparedness and 
management of public health/biological events in hospitals and 
primary health clinics. 

MATERIALS AND METHODS 
STUDY DESIGN 

A joint project, aimed to develop an evaluation tool for biological 
preparedness, was initiated under the framework of the "Research 
of Civil Security," supported by the German Federal Ministry of 
Education and Research and the Israeli Ministry of Health. A joint 
German-Israeli steering committee was established consisting of 
12 members, experts in emergency management, communica- 
ble diseases, public health, and/or healthcare management, as 
well as software development. The committee worked as a team 
throughout the development process that lasted from April 2010 
to September 2013. 

A comprehensive literature review of publications from 1995 
to 2012 was conducted using PubMed and Google Scholar, based 
on the following keywords: biological events, communicable dis- 
eases, pandemics and epidemics, emergency preparedness, emer- 
gency management, emergency response, measuring emergency 
preparedness, and evaluation tools. An in-depth analysis of each 



of the identified articles was implemented in order to extract 
components relevant to biological events. The components were 
classified to categories relevant to manage biological events, and 
were then transformed into measurable parameters that can be 
integrated in an evaluation tool. 

SELECTION OF PARTICIPANTS 

In the framework of a modified Delphi process (17), the parame- 
ters were disseminated to 228 content experts, 188 from Germany 
and 40 from Israel. The experts were from various fields of public 
health and emergency management, including healthcare sys- 
tems' managers, front line clinicians, medical professionals, first 
responders, and infectious diseases specialists. 

INTERVENTIONS 

Two modified Delphi cycles were conducted to achieve a consensus 
of 75% or higher regarding the evaluation parameters. The content 
experts were requested to mark their agreement or disagreement to 
include parameters in the tool, and rank each parameter accord- 
ing to its level of importance. Three levels of importance could 
be defined - very important (level 1), important (level 2), or less 
important (level 3). 

The Delphi process was run electronically on a protected 
Internet platform that enabled the respondents to mark their 
opinions directly in the web-based tool (http://www.delphi- study, 
com). Most experts were requested to comment on the relevance 
and importance of 30-50 parameters each. Fifteen experts were 
requested to evaluate and rank all 188 parameters. 

Parallel to the modified Delphi cycles, indicators were inte- 
grated in the evaluation tool, describing the level of performance 
of all elements included in each parameter. Each indicator was 
designed to allow the evaluator to objectively attest to the level of 
adherence to the parameter, avoiding the need to base the reported 
finding on a verbal description provided by the evaluated facility's 
representative. The indicators were classified to four potential lev- 
els: satisfactory (the aspired benchmark); minor revisions needed; 
major revisions needed; and, not satisfactory (unacceptable level 
of performance). 

In order to ensure simplicity and objectivity in utilizing the 
tools, two additional elements were integrated in the evalua- 
tion tool for each parameter: (1) description of the source of 
information/methodology of assessment; and, (2) explanation/full 
description concerning the appropriate implementation of the 
parameter in the medical facility. 

Following the Delphi cycles, a focus group study was conducted 
between March 2012 and July 2013, to identify the psychosocial 
needs of the hospital staff in a biological event, in order to inte- 
grate them in the evaluation tool. Groups of two to six people 
consisting of different professionals in the hospital (medical as 
well as logistic personnel) were presented with a SARS-like sce- 
nario. They were asked to discuss the question: "In a situation 
like this: what does someone in your position need in order to be 
able to work professionally and feel safe?" A qualitative data col- 
lection method was used to maintain openness for unanticipated 
emerging themes (18). 

The focus groups were recorded and transcribed verbatim. 
Based on the thematic framework approach by Ritchie et al. (19), 
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the data were analyzed and edited. Results from the analysis were 
added to the evaluation tool. 

Following the feedback from the hospitals' focus groups, the 
tool was again revised. 

Exercises in six hospitals were conducted to review the applica- 
bility of the evaluation tool to their needs. The scenario simulated 
one index patient accompanied by one contact person that pre- 
sented to the emergency department of the hospital, arriving a few 
days after their return from a 2-week vacation in Turkey. While 
the contact was symptom-free, the index patient was bitten by a 
tick, suffering from flu-like symptoms, gum bleeding, and nausea 
caused by an unidentified biological agent. 

Prior to the exercise, the hospitals' administrations received 
a beta-version of the evaluation tool in order to implement a 
self-evaluation using the core parameters. They were requested 
to submit the results to the steering committee within a period 
of 2-7 weeks, along with suggestions for corrections and com- 
ments. Following review of the tool, the hospitals participated in 
a table-top as well as a practical exercise. 

The table-top exercise was dedicated to a presentation of the 
hospital's plan to respond to the described scenario. In the practical 
exercise, actors simulated the scenario at the emergency depart- 
ment. The exercise was completed within 2 h, when an ambulance 
for infectious patients virtually arrived to transfer the index patient 
to an isolation unit at the University hospital in Frankfurt on the 
Main, Germany. In agreement with the hospitals, questionnaires 
were distributed in order to have a standardized evaluation of the 
exercise. Approximately 10 observers participated in each exercise, 
consisting mainly of members of the steering committee. Follow- 
ing the exercises, the hospital teams were requested to analyze their 



performance in order to adjust their institutional plans, review 
the evaluation tool, and comment on whether modifications were 
needed. 

OUTCOMES 

Based on the results that were achieved in the modified Delphi 
cycles, focus groups, table-tops, and exercises, the evaluation tool 
was revised. Only parameters that reached the minimum level of 
consensus (>75%) were integrated in the tool. The overall level of 
importance of each parameter was calculated based on the median 
ranking that was achieved in the second Delphi cycle. 

A beta-version of a web-based computerized system was devel- 
oped, aimed at enabling each hospital or community clinic inde- 
pendent use of the evaluation tool. The system was designed for 
use by both the individual medical facility and/or a governing 
authority, such as the management of a cooperation of hospitals 
and clinics or the Israeli Ministry of Health. 

ANALYSIS 

In order to generate the weight of each parameter, an algorithm 
with the following data was used: the importance of the category, 
the value of the parameter according to the level of importance 
(very important — > value 10, important^- 5, less important — > 1), 
and the number of parameters in each category. 

The methodology for developing the evaluation tool is pre- 
sented in Figure 1 . 

RESULTS 

One hundred articles were identified as relevant to the scope of 
the study - emergency management of a biological event. The 
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FIGURE 1 | Methodology for developing the evaluation tool 
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various subjects discussed in the articles were classified into five 
categories: policy and planning, medical management, personnel, 
communication, and infrastructure. The components of emer- 
gency management from the five categories were analyzed and 
then portrayed to define 188 measurable parameters. 

CHARACTERISTICS OF STUDY SUBJECTS 

The parameters were disseminated to 228 content experts; 188 
from Germany and 40 from Israel. Overall response rate for the 
first Delphi cycle was 51.3% (117 experts). Comparison of replies 
from Germany and Israel presented response rates of 46.8% (88 
experts) and 72.5% (29 respondents), respectively. The desired 
level of consensus (>75%) was achieved for most of the parame- 
ters; after the first round 176 out of 188 parameters were rated 
by more than 75% of the content experts for inclusion in the 
evaluation tool. In addition, 1,031 comments and suggestions 
were given by the experts, which provided an opportunity to 
portray the decision-making considerations in the next Delphi 
cycle. 

Despite the high consensus levels achieved in the first cycle, 
the ratings as well as the comments of the experts were aggregated 
and disseminated to the respondents for a second modified Delphi 
cycle. Overall response rate for the second modified Delphi cycle 
was 55.6% (65 respondents out of 117), based on 45.5% (40 out of 
88 experts) from Germany and 86.2% (25 out of 29 experts) from 
Israel. 

MAIN RESULTS 

One hundred eighty-three parameters reached the desired level of 
consensus (>75%) in the second Delphi cycle and were then inte- 
grated in the evaluation tool. Two hundred nine comments and 
suggestions were given in the second cycle. The level of consensus 
regarding the parameters in the two modified Delphi cycles are 
presented in Table 1. 

Comparison of the views of the German versus Israeli content 
experts, as expressed in the two modified Delphi cycles, presents 
a strong similarity concerning elements that should be integrated 
in the evaluation tool. Overall, 91% of the experts from Germany 
versus 94% of the respondents from Israel expressed their sup- 
port in integrating the 183 parameters in the evaluation tool. The 
level of consensus from both countries concerning each category 
is presented in Figure 2. 



Table 1 | Level of consensus in the two modified Delphi cycles. 



Category 


Total 


Delphi cycle 


Delphi cycle 




number of 


1 - % of 


II - % of 




parameters 


parameters 


parameters 






>75% 


>75% 






consensus 


consensus 


Policy and planning 


56 


91% (51) 


100% (56) 


Medical management 


38 


95% (36) 


97% (37) 


Personne 


46 


89% (41) 


91 % (42) 


Communication 


29 


100% (29) 


100% (29) 


Infrastructure 


19 


100% (19) 


100% (19) 


Total 


188 parameters 


176 parameters 


183 parameters 



In addition, the parameters were classified based on the charac- 
teristics of the hospitals in regard to the competence and respon- 
sibility concerning infectious diseases (module). Three modules 
were defined: (1) core (hospitals without specialization in infec- 
tious diseases and/or infectious disease consultants); (2) inter- 
mediate (hospitals with isolation facilities able to treat a limited 
number of patients with highly infectious diseases); (3) advanced 
(hospitals responsible for management of large numbers of highly 
infectious disease patients). 

In order to rank the relative importance of each of the five 
categories, a mini-Delphi was conducted among the members 
of the steering committee (n = 9) and the mean was calculated. 
The relative importance of the categories was determined as fol- 
lows: medical management - 25.6%; personnel - 25%; policy and 
planning - 20.6%; infrastructure - 15.6%; and, communication - 
13.2%. These data were further used to calculate the overall weight 
of each parameter in the context of its category by using the fol- 
lowing formula: weight = importance of category x value/no. of 
parameters in category. 

One focus group was initiated in Israel, consisting of five mem- 
bers of the Biological National Advisory Committee, amongst 
them three physicians and one support team member (all males), 
and one female nurse. Eleven focus groups were conducted in 
Germany (consisting of staff from six hospitals). The participants 
included 54 professionals (36 nurses, 9 physicians, and 9 support 
team members), among them 17 men and 37 women, consisting 
of two to six participants in each meeting. Emerging themes in the 
discussions focused mainly on information flow, problems with 
hierarchy, organization and leadership in case of an event, and 
personal protective equipment. 

One additional element was found by the focus groups in Ger- 
many to be lacking in the evaluation tool. After analysis of the data 
raised in the focus groups, further information could be provided 
as auxiliary knowledge. Thus, 1 parameter and 40 supplements 
(recommendations, additional information, or suggestions as to 
how to solve common problems) were suggested for inclusion. No 
changes to the evaluation tool were recommended by the Israeli 
focus group. 

Four hospitals in Hesse (HS) and four in Rhineland-Palatinate 
(RLP) were recruited in Germany for the exercises. Two of the four 
hospitals in HS belonged to the same agency and in RLP, three of 
the four were run by the same administration. 

The evaluation tool was finalized after being modified accord- 
ing to the comments and suggestions recommended in the Delphi 
cycles, discussions within the steering committee, input received 
from the focus groups and feedback from the hospital exercises. 
The result was deletion of 1 1 parameters from the evaluation tool, 
due to a strong similarity to other parameters (7 parameters), cur- 
rent lack of technology (one parameter), or because they fall under 
the responsibility of the public health services rather than the hos- 
pitals or clinics (4 parameters). In addition, one parameter was 
added from the focus group study, thus the final evaluation tool 
consisted of a total of 172 parameters. 

The median level of importance of each parameter was calcu- 
lated based on the ranking recommended by the content experts 
in the Delphi process. As one parameter originated from the focus 
groups, it was not ranked in the two Delphi cycles. Therefore, the 
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steering committee (n = 6) voted for its importance. Twenty- four 
parameters were ranked as very important, 120 as important, and 
28 as less important. Classification of the parameters in each cate- 
gory according to their importance in the finalized evaluation tool 
is presented in Table 2. 

The final version of the evaluation tool is structured as follows: 
category, subcategory, module, functionary, area of operation, 
parameter, level of importance, source of information/evaluation 
technique, four levels of performance, and additional explana- 
tion. The evaluation tool includes 172 parameters, classified to 5 
categories and 15 subcategories. An example of the parameters 
and indicators integrated in the evaluation tool is presented in 
Table 3. 

A web-based software tool was developed for use during the 
evaluation process. The tool is structured in three tiers to allow 
non-complicated utilization by the different users, as well as flex- 
ibility in integrating changes. New applications or content can 
easily be developed, without need for major changes, thus a contin- 
uous update and maintenance are assured. ASP- NET technology 
from Microsoft was used for the web application and the data base 
system is Microsoft SQL server. Taking into consideration the dif- 
ferent clients, the system was localized in three languages - English, 
German, and Hebrew. The software service enables to record level 
of performance regarding each parameter and can then produce 
a report of the level of preparedness of the facility. The report 
includes scores of the emergency preparedness according to the 
different categories of emergency response for biological events 
and the relevant scaling (rank) of each parameter, as well as a 
list of all elements that were found as needing modifications or 
performed in an unsatisfactory way. 



Table 2 | Classification of parameters in the final evaluation tool 
according to level of importance. 



Importance 



Total 



Very 



Important Less 



important 




important 




8 


37 


5 


50 


3 


23 


10 


36 


4 


28 


9 


41 


6 


19* 


2 


27 


3 


13 


2 


18 


24 


120 


28 


172 



Category 



Policy and planning 

Medical management 

Personnel 

Communication 

Infrastructure 

Total 



'Including one parameter as a result of the focus groups. 
LIMITATIONS 

A few limitations should be noted with respect to the bilateral 
study. The parameters and indicators were initially developed in 
English and then translated to German and Hebrew, respectively. It 
is possible that some of the meanings were distorted in the trans- 
lation process. In effort to overcome this risk, the joint steering 
committee reviewed each parameter several times, checking the 
translation from English to the two respective languages and its 
re-translation back to English. The Delphi cycles in Israel were con- 
ducted in English rather than in Hebrew; this might have impacted 
on the response rates. 

The experts participating in the Delphi cycles were asked about 
the relevance of parameters but not for completeness of content. 
It cannot be excluded that certain parameters might have been 
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approved, though the content was not offered by the steering 
committee. To overcome this possibility, apart from the compre- 
hensive literature review, hospital exercises and focus groups were 
conducted. 

For further evaluation, the data protection laws in Germany 
limit the ability to conduct comprehensive comparisons of results 
concerning levels of emergency preparedness of each medical 
facility. 

DISCUSSION 

An outbreak of an infectious disease may cause significant dam- 
age to any society, and has the potential to rapidly spread and 
impact countries worldwide (1, 2). Considering the severe risk, 
effective preparedness and management of such an event are cru- 
cial components of emergency response (7). Assuring an effective 
response for biological threats necessitates creation and sustenance 
of a reliable and valid evaluation process that can be implemented 
continuously (12, 13). 

Various tools are utilized globally to evaluate readiness of hos- 
pitals and other medical facilities to manage emergency scenarios 
(11, 16). Nonetheless, these tools are not suitable for biological 
events (2, 9, 11), due to the unique characteristics of disease out- 
breaks (incubation periods, gradual development of symptoms, 
need for surveillance systems to monitor morbidity and mortality, 
challenge of detection and identification, specific pharmaceuticals, 
particular protective gear, isolation capabilities, etc.). 

The BEPE project initiated jointly by Germany and Israel 
under the framework of the "Research of Civil Security" was 
designed to develop such an evaluation tool that will enable to 
identify strengths and weaknesses in the preparedness system 
and improve gaps. The tool is targeted to save lives upon an 
occurrence of a public health threat/biological event. The para- 
meters integrated in the evaluation tool serve as benchmarks 
for the healthcare facilities that delineate actions that need to 
be implemented in order to achieve preparedness for biological 
events (20). The parameters enable both the governing author- 
ities and each facility to independently evaluate the level of 
readiness. 

Development of clear and well-defined indicators is a vital com- 
ponent of any evaluation tool. Indicators are aimed to clearly 
present the level of performance of each parameter. They pro- 
vide focus and direction to help identify strengths and gaps in 
the emergency-preparedness levels and facilitate target quality 
improvement efforts (11, 21). As recommended in other stud- 
ies (22), the indicators in the present tool were developed through 
the adoption of the SMART criteria (specific, measurable, achiev- 
able, relevant, and time-bound). Through the modified Delphi 
process, validity concerning the parameters was achieved as well 
as promotion of inter-rater reliability thus enabling to draw con- 
clusions regarding the effectiveness of the medical institutions 
to manage biological events (23). The indicators are classified 
to four levels, in order to encourage improvement. Consider- 
ing the complexity, needed resources and efforts that must be 
invested in each activity designed to increase preparedness, inte- 
gration of less than four levels of performance may cause inabil- 
ity to witness a positive trend of correcting gaps. Providing the 
opportunity to display even small advancements empowers the 



health facilities' administrations in their efforts to achieve quality 
improvement. 

The high resemblance of the responses of the German ver- 
sus the Israeli experts in the modified Delphi process supports the 
perception that preparedness for biological events is similar world- 
wide (6, 8). Characteristics of Germany and Israel differ in many 
aspects such as size (populations of 81.5 versus 8 million, respec- 
tively); type of government (Israel is characterized by a centralized 
government while Germany is a federation of states); responsibil- 
ity of each hospital regarding biological events (in Germany there 
are three tiers of hospitals classified according to preparedness for 
a single patient, intermediate or mass casualty events while Israeli 
hospitals are all required to be ready for mass casualties). The 
evaluation tool is targeted in Germany for the independent use 
of each medical facility, while in Israel it is targeted not solely for 
the facility's administration but also to the Ministry of Health 
and other governing authorities. Nevertheless, experts of both 
countries found the parameters included in the evaluation tool 
appropriate and relevant for biological preparedness, thus signi- 
fying that the developed tool can most probably be adopted as 
a generic tool. Adoption of such an ongoing means of quality 
improvement mechanism is crucial to all societies (24) and may 
contribute to creating and maintaining capacities to respond to 
potential biological events. Implementation of such mechanisms 
in multiple systems may enable to identify global trends and thus 
increase public health and safety. 

In summary, infectious and communicable diseases may impact 
on the public health of the world's population; therefore it is 
of global interest to maintain a continuous high level of alert 
and emergency preparedness to manage such events. An ongo- 
ing evaluation means, such as the tool that was developed in 
the present study, may facilitate the need for a valid and reliable 
mechanism that can be widely adopted and implemented as part 
of quality assurance measures. The developed tool is based on 
measurable parameters and indicators that can effectively present 
strengths and weaknesses in managing a response to a public health 
threat, and accordingly, steps can be implemented to improve the 
readiness. Adoption of such a tool is an important component 
of attaining and assuring public health and effective emergency 
management. 
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